Topic model and document-clustering evaluations either use automated metrics that align poorly with human preferences or require expert labels that are intractable to scale. We design a scalable human evaluation protocol and a corresponding automated approximation that reflect practitioners' real-world usage of models. Annotators -- or an LLM-based proxy -- review text items assigned to a topic or cluster, infer a category for the group, then apply that category to other documents. Using this protocol, we collect extensive crowdworker annotations of outputs from a diverse set of topic models on two datasets. We then use these annotations to validate automated proxies, finding that the best LLM proxies are statistically indistinguishable from a human annotator and can therefore serve as a reasonable substitute in automated evaluations. Package, web interface, and data are at https://github.com/ahoho/proxann

Today's post comes to us from Workforce Institute board member, Dan Schawbel. Here he wonders where do human fit in an AI world? Over the past few years, I've been exploring the relationship between humans and technology in the workplace. The big question that everyone wants to know the answer to is, "what is a human's role in an AI-driven world?" We all want to know what it means to be a human when technology seems to be consuming everything around us.

Episode 106 – Edward Hess discusses The New Smart Machine Age and where humans fit in. Will smart machines and robots start taking our jobs? What can we do to prepare for the inevitable transformation to robotics and automation? Ed is a Professor of Business Administration at the Darden Graduate School of Business at the University of Virginia. You've seen and heard him in places like WSJ Radio, CNBC, NPR, and Investor's Business Daily.

February 8, 2017 --Someday soon, you will ask a robot to fetch a slice of pizza from your refrigerator. On that day, you'll trust that the robot won't tear through your walls and rip the fridge door off its hinges to get at your leftovers. Getting robots to do the things humans do in the ways that humans do them (or better) without human intervention is an immensely wicked problem of autonomy. With as many as half of American jobs at risk of automation according to one study, and with an expected 10 million self driving cars on the road by 2020, robots are going to be everywhere, forever, and they won't go away. The enormous scope and scale of how autonomous robots will begin changing our lives requires the public and technologists alike to consider the challenges of autonomy.

Robin Hanson thinks the robot takeover, when it comes, will be in the form of emulations. In his new book, The Age of Em, the economist explains: you take the best and brightest 200 human beings on the planet, you scan their brains and you get robots that to all intents and purposes are indivisible from the humans on which they are based, except a thousand times faster and better. For some reason, conversationally, Hanson repeatedly calls these 200 human prototypes "the billionaires", even though having a billion in any currency would be strong evidence against your being the brightest, since you have no sense of how much is enough. But that's just a natural difference of opinion between an economist and a mediocre person who is now afraid of the future. These Ems, being superior at everything and having no material needs that couldn't be satisfied virtually, will undercut humans in the labour market, and render us totally unnecessary.

Robin Hanson thinks the robot takeover, when it comes, will be in the form of emulations. In his new book, The Age of Em, the economist explains: you take the best and brightest 200 human beings on the planet, you scan their brains and you get robots that to all intents and purposes are indivisible from the humans on which they are based, except a thousand times faster and better. Related: The Guardian view on artificial intelligence: look out, it's ahead of you Editorial For some reason, conversationally, Hanson repeatedly calls these 200 human prototypes "the billionaires", even though having a billion in any currency would be strong evidence against your being the brightest, since you have no sense of how much is enough. But that's just a natural difference of opinion between an economist and a mediocre person who is now afraid of the future. These Ems, being superior at everything and having no material needs that couldn't be satisfied virtually, will undercut humans in the labour market, and render us totally unnecessary.

Robin Hanson thinks the robot takeover, when it comes, will be in the form of emulations. In his new book, The Age of Em, the economist explains: you take the best and brightest 200 human beings on the planet, you scan their brains and you get robots that to all intents and purposes are indivisible from the humans on which they are based, except a thousand times faster and better. Related: The Guardian view on artificial intelligence: look out, it's ahead of you Editorial For some reason, conversationally, Hanson repeatedly calls these 200 human prototypes "the billionaires", even though having a billion in any currency would be strong evidence against your being the brightest, since you have no sense of how much is enough. But that's just a natural difference of opinion between an economist and a mediocre person who is now afraid of the future. These Ems, being superior at everything and having no material needs that couldn't be satisfied virtually, will undercut humans in the labour market, and render us totally unnecessary.